https://www.fsbpt.org RSS feeds for FSBPT 60 https://www.fsbpt.org/News-Events/Articles/articleType/ArticleView/articleId/1/Setting-a-Passing-Score#Comments 0 https://www.fsbpt.org/DesktopModules/DnnForge%20-%20NewsArticles/RssComments.aspx?TabID=226&ModuleID=1673&ArticleID=1 https://www.fsbpt.org:443/DesktopModules/DnnForge%20-%20NewsArticles/Tracking/Trackback.aspx?ArticleID=1&PortalID=0&TabID=226 https://www.fsbpt.org/News-Events/Articles/articleType/ArticleView/articleId/1/Setting-a-Passing-Score Forum, Volume 21, Number 2 What Does the NPTE Passing Score Reflect? Setting a passing score involves the process by which a performance standard is established. The passing score typically reflects the test score corresponding to a desired level of performance and is used for making decisions about what level of performance is high enough for a given purpose. The passing scores established for the National Physical Therapy Licensure Examinations (NPTE) reflect the level of performance required to provide minimally safe and competent physical therapy services by physical therapists and physical therapist assistants. Individuals scoring at or above the passing score have met the performance standard and are eligible for licensure, and individuals scoring below the passing score have failed to meet the performance standard and are not yet eligible for licensure. The term “passing score” is used interchangeably with the terms cut score and performance standard. Passing Scores Do Not Exist Independently There really is no passing score that exists independently. Rather, the passing score is the product of a deliberative process of the experience and the judgments of people who are qualified to make those judgments. Setting a passing score is a policy decision related to the mission of organization, in this case, to ensure that the public is protected from practitioners who are not competent. On the other hand, one must make certain the bar is not so high that it prohibits individuals who do have the necessary knowledge and skills from being able to enter the profession. Numerous Steps are Taken to Ensure That the Score is Defensible Establishing a performance standard is critical because of its impact on individuals, and numerous steps must be taken to ensure that the score is defensible. The Standards for Educational Psychological Testing contains criteria for establishing defensible passing scores, which have been endorsed by many organizations as the standard for practice both in terms of defensibility and best practices in our field. Included among the criteria are the qualifications of individuals participating in the setting of performance standards, the procedures by which performance standards are set, and the efforts made to validate the results of the standard setting process on an ongoing basis. All of these criteria combined are intended to ensure that the process is documented and defensible. Establishing a passing score, including standards for passing the NPTE, involves six steps. Deciding on a method that is appropriate for the test at hand. Selecting qualified participants (individuals in the profession considered well qualified in their practice). Training participants on the standard-setting method. Providing feedback about participants’ judgments in carrying out the method. Calculating a passing score. Gathering validity evidence that is intended to bear on the question, “Do the folks who are awarded the license or the credential seem to be the kinds of folks who actually possess the kinds of characteristics important for practice?” Choosing a method Historically, the Federation has relied on criterion-referenced methods for setting passing scores since the early 1990s, particularly the modified-Angoff method.1 This criterion-referenced method asks, “What are the levels of knowledge and skill that are required?” and sets the bar at that level. If every examinee had that level of knowledge or skill, the pass rate would be 100%. The modified-Angoff technique is probably the most widely used method in health professions today, and it probably has the largest research base. Because of its ease of use and large research base, this method also is considered best practice. Selecting participants The next step is to select qualified participants who are representative of the profession. Participants are selected to be representative of the profession in terms of practice setting, specialty, geographic location, race and gender. Participants also are required to be knowledgeable about how the NPTE is developed. Training Participants The essential background required for the Angoff method is deliberation, discussion and formulation of a concept of a minimally competent or borderline level of knowledge or skill. For example, the statement reflecting minimal competence on the NPTE for physical therapists is “the minimal knowledge, judgment, technical, and interpersonal skills required to safely practice physical therapy.” It includes skills and knowledge on examination, evaluation, diagnoses, prognosis, intervention, and outcome assessment. As part of their training, the panel is required to develop what is called a “key conceptualization” of the minimally qualified candidate – the hypothetical line between competent for entry into practice and not competent for entry into practice. What does that person look like? What can they do? What can they not do? Where is point you want to create a passing score between competent and not competent? Rating Test Item Difficulty After the panel is trained in using the modified-Angoff method and has developed key conceptualizations, panelists are asked to provide a rating of test items - the proportion of minimally competent examinees that would be expected to get that item correct. This judgment is made on each test question included on a test form. Feedback After initial ratings are made, panelists are encouraged to discuss their individual ratings. The raters consider the divergence, the heterogeneity, and the variability in their ratings, asking of themselves and other panelists: “Why did you rate it that way? Who gave this one such a high rating or a low rating? Why did that happen?” Through this discussion, panelists see if they are able to come to a consensus in their judgments. However, consensus isn’t required, it is just facilitated. In addition to feedback on other panelists' ratings, the panel may receive feedback on the proportion of minimally-competent test takers who answered each question correctly in the past, as well as the impact of these standards on the pass rate. Setting New Scores in 2007 The passing score must be revisited periodically to ensure that it is responsive to changes in practice. At a minimum, the passing score is revisited whenever the scope of practice changes and the NPTE test is revised. For the NPTE, this process of reviewing the scope of entry-level practice, updating the test and revisiting the passing score occurs at least every five years. Passing scores were set for the NPTE in 2002 after the test had been updated as a result of a regular analysis of practice. The passing score for the NPTE for physical therapists was revisited in 2005 as a result of ongoing review. The FSBPT is currently conducting an analysis of entry-level practice for physical therapists and physical therapist assistants, and the results of this analysis of practice will be used to update the test. New test forms will be assembled beginning in 2007 and new passing scores will be established in fall 2007. After that process ends, the final passing score is recommended to the Federation Board of Directors. The Board of Directors will review the whole process, asking questions such as: Was the process implemented correctly? Was it done according to professional standards? Were there glitches? Were there modifications that were unexpected? After a thorough review, the Board of Directors will make its decision regarding the recommendation. If approved, it becomes the new passing score. Finally, new forms of the NPTE and new passing scores will be launched March 1, 2008. 1 In 2005, the Federation used an alternative criterion-referenced method to review the passing score for the NPTE for physical therapists called the Direct Consensus method. SuperUser Account Tue, 29 Jan 2013 22:17:00 GMT f1397696-738c-4295-afcd-943feb885714:1 https://www.fsbpt.org/News-Events/Articles/articleType/ArticleView/articleId/2/Developing-Content-Validity-Practice-Analysis-to-Test-Content-Outline#Comments 0 https://www.fsbpt.org/DesktopModules/DnnForge%20-%20NewsArticles/RssComments.aspx?TabID=226&ModuleID=1673&ArticleID=2 https://www.fsbpt.org:443/DesktopModules/DnnForge%20-%20NewsArticles/Tracking/Trackback.aspx?ArticleID=2&PortalID=0&TabID=226 https://www.fsbpt.org/News-Events/Articles/articleType/ArticleView/articleId/2/Developing-Content-Validity-Practice-Analysis-to-Test-Content-Outline Forum, Volume 21, Number 1 What types of knowledge need to be covered on the National Physical Therapy Examination (NPTE) in order to determine if an individual is minimally competent to work as an entry-level PT or PTA? It's a key question that ultimately is answered through a process referred to as a practice analysis. A practice analysis is a study that systematically determines these responsibilities or "activities," and the knowledge and skill requirements (KSRs) for performing the activities. The results of a practice analysis are useful for informing decisions about the test content outline and for providing evidence of content validity. Purpose of a Practice Analysis FSBPT uses practice analysis to: Verify current entry-level practice in the physical therapy profession; Determine the knowledge and skill requirements (KSRs) required to perform at an entry level; and Maintain the content validity of the NPTE by ensuring that the test content outline continues to measure entry-level knowledge and skills important for public protection. The practice analysis conducted by the FSBPT is not intended to encompass the entire physical therapy profession or what it “should be.” It also is not intended to be a curriculum outline or synopsis of physical therapy education, an evaluation of advanced knowledge and skills for physical therapy, or to express an opinion or a position on physical therapy. A Standard Approach The practice analyses conducted by the Federation reflect best practices and meet criteria set forth in the Standards for Education and Psychological Testing developed by the American Education Research Association, the American Psychological Association and the National Council on Measurement in Education for defensible methods of establishing content validity in licensure and certification testing. The frequency with which a practice analysis should be conducted depends on how rapidly job requirements change; for the physical therapy profession, practice analyses are typically conducted every five years. Previous and Future Practice Analyses The Federation conducted a study in the United States and Canada in 1995-1996, with individual content outlines for PTs and PTAs in each country. The last study in the United States was completed in 2002. The Federation has just begun its 2006 Analysis of Practice of physical therapists and physical therapists assistants. Overview of a Practice Analysis Job responsibilities differ for PTs and PTAs. As a result, a different examination is developed for each occupation. The practice analyses for physical therapist and physical therapist assistant NPTEs also are parallel but separate processes. The Federation first develops and pilots surveys to obtain information on activities of entry-level PTs and PTAs and the knowledge and skill requirements (KSRs) for these activities. The pilot surveys are then revised and distributed to a nationally representative sample of PTs and PTAs. To the Federation, that means sending surveys to physical therapists and physical therapist assistants in all 53 licensure jurisdictions. The survey data are analyzed to determine the set of activities and KSRs critical for entry-level practice. The results are used to update the PT and PTA NPTE test content outlines, which ensures that the tests continue to measure important information in the right proportions. Phases of a Practice analysis Conduct a literature review of current physical therapy practice documents to develop a list of potential activities and KSRs for the surveys Develop and pilot surveys for entry-level PT and entry-level PTA activities and KSRs (knowledge and skill requirements) Develop final surveys based on data from pilot surveys Distribute final surveys to a nationally representative sample of PTs and PTAs Analyze survey data to determine critical activities Link critical activities to KSRs Use the findings to update the content outlines for the PT and PTA NPTE The Oversight Panel and Task Forces The Federation solicits nominations from member jurisdictions and professional physical therapy groups and sections in order to ensure the oversight panel and task forces represent the profession in terms of physical therapy practice settings, ethnicities, ages, lengths of practice and regions of the United States. The Practice Analysis Oversight Panel is appointed to oversee the practice analysis process and task forces. Members of the panel are familiar with the NPTE development process, its content outlines and current practice issues in physical therapy. The panel reviews the pilot surveys, activity lists and KSRs, providing advice and guidance to the task forces and staff throughout the process. Two task forces are appointed, one for the PT practice analysis and one for the PTA practice analysis. They produce the pilot and final surveys and use the survey results to build new content outlines for the PT and PTA examinations. The task forces participate in two assignments: (1) developing lists of job activities and knowledge and skill requirements; and (2) reviewing preliminary survey and linkage results. The results of those assignments are used to update the PT and PTA test content outlines. Survey Development Both pilot and final surveys ask for demographic and professional background information from the individual taking the survey. They include lists of activities and KRS, which will be rated on scales to assess their importance to entry-level practice. Typical scales include: 1) acquisition level, 2) consequence of incorrect performance, and 3) frequency of performance. The pilot survey also asks general questions about the survey itself such as "Did we forget an activity that was essential?" and "Was some activity not physical therapy?" Acquisition Level: At what level of practice are the knowledge requirements and skills necessary to independently perform this activity typically required? Consequence of Incorrect Performance: When considering the risk of unnecessary complications, impairment of function or serious distress to patients, how much physical or psychological harm will the incorrect performance of this activity most likely cause the patient? Frequency of Performance: How often do you perform this activity? Task force members review the pilot survey findings and revise the survey as suggested by the respondents. Revisions included clarifying confusing areas, condensing the survey and including the amount of time needed to complete the survey in the request to participate. Survey Participation The survey of activities is representative, meaning that the number of pilot and final survey participants are based on the total number of PTs and PTAs in jurisdictions. The ideal survey subjects have up to five years of experience so they are not too far from entry-level practice. They represent the same diversity as the profession in key areas such as gender, age, ethnicity, region and various clinical settings. All have passed the exam and are licensed or certified. A sophisticated distribution plan is used to elicit as many responses as possible. For instance, an alert letter may be offered the survey on a web link. Non-respondents receive a follow-up letter with a paper survey. This communication may be followed up with a postcard, then a second follow-up letter with another paper survey. The typical respondent to the last practice analysis survey (done in 2000) was female, white, licensed or certified between 1996 and 2000, full-time/salaried, working in direct patient care (especially ambulatory/outpatient and acute care) and more likely to report obtaining an MPT or MSPT credential. Survey Analysis Ratings from the activity survey are combined into a single index of criticality, with the entry-level activities having consequence for public protection receiving the most weight. Knowledges and skills required to perform these important entry-level activities are linked to the critical activities and then structured into a preliminary content outline for each exam. The task forces are reconvened to review and finalize the test content outlines. It should be noted that activities may be dropped from a content outline if they are part of advanced practice or too infrequently performed to warrant inclusion. Once the test content outline is finalized, new forms of the NPTE are assembled to meet the updated test content outlines. A Status Report on the 2006 Practice Analysis To date, the Practice Analysis Oversight Panel and task forces have been appointed. They have generated a list of activities and KSRs, which are currently being used to develop the pilot surveys. The pilot surveys will be sent in late spring this year and the national survey will be sent this summer. We expect to finalize test content outlines by early fall. Mon, 28 Jan 2013 15:33:00 GMT f1397696-738c-4295-afcd-943feb885714:2